Physician quality scoring

ABSTRACT

Aspects of the present invention relate to system and methods for assigning quality scores to one or more caregivers, such as physicians. In embodiments, a ranking or score may be based, at least in part, upon a combination of quality scores from one or more stages in a physician&#39;s academic training and clinic practice and based, at least in part, upon the quality of peers of that physician at various stages in the career progression of the physician. This information may be used to help a potential patient identify a physician for their care.

BACKGROUND Field of Invention

The present invention relates generally to data processing, and relatesmore particularly to system and methods for assessing and scoring aphysician.

Description of the Related Art

Healthcare as an industry has become increasing more complex and costly.The number and type of healthcare providers available to patients islikewise vast. Added to this ever-increasingly expanding system is asignificant absence of important information. Unlike most otherindustries, the healthcare industry provides very little information tohelp patients make informed decision when selecting a physician. Yet,the selection of a physician by a patient can have considerable—evencritical—effects upon the patient's treatment and recovery.

Currently, most reviews or rankings of physicians, particularly thosefrom patients, are based on non-quality-related factors, such asniceness of doctor, wait times, cleanliness of the waiting area, etc.Unfortunately, this information is of little or no value when trying tofind the best quality doctor and can, in fact, be misleading anddetrimental if the wrong metrics are taking for surrogates for quality.

Accordingly, what is needed are systems and methods to help gather datarelated to physicians and use that data to help assess the quality of acaregiver or set of caregivers.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will be made to embodiments of the invention, examples ofwhich may be illustrated in the accompanying figures, in which likeparts may be referred to by like or similar numerals. These figures areintended to be illustrative, not limiting. Although the invention isgenerally described in the context of these embodiments, it should beunderstood that it is not intended to limit the spirit and scope of theinvention to these particular embodiments. These drawings shall in noway limit any changes in form and detail that may be made to theinvention by one skilled in the art without departing from the spiritand scope of the invention.

FIG. 1 depicts various stages in a physician's career that may be usedin scoring a physician for a patient's specific needs according toembodiments of the present invention.

FIG. 2 depicts relationships between medical schools, hospitals, andspecialty programs for which residency and fellowship programs may beoffered according to embodiments of the present invention.

FIG. 3 graphically depicts the relationships between peers and medicalschool rating according to embodiments of the present invention.

FIG. 4 graphically depicts the relationships between peers and residencyprogram rating according to embodiments of the present invention.

FIG. 5 graphically depicts the relationships between peers andfellowship program rating according to embodiments of the presentinvention.

FIG. 6 depicts a methodology for assigning an overall training score toa physician according to embodiments of the present invention.

FIG. 7 depicts a methodology for determining a correlation factor aspart of the iteration process according to embodiments of the presentinvention

FIG. 8 depicts a method for assigning a physician's overall trainingusing correlations according to embodiments of the present invention.

FIG. 9 graphically depicts the relationships between peers and practicelocations/groups according to embodiments of the present invention.

FIG. 10 depicts a method for determining a physician's overall qualityscore according to embodiments of the present invention.

FIG. 11 depicts a block diagram of an exemplary information handlingsystem node according to embodiments of the present invention.

FIG. 12 depicts a block diagram of one or more sets of datastoresaccording to embodiments of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, for purposes of explanation, specificexamples and details are set forth in order to provide an understandingof the invention. It will be apparent, however, to one skilled in theart that the invention may be practiced without these details.Well-known process steps may not be described in detail in order toavoid unnecessarily obscuring the present invention. Other applicationsare possible, such that the following examples should not be taken aslimiting. Furthermore, one skilled in the art will recognize thataspects of the present invention, described herein, may be implementedin a variety of ways, including software, hardware, firmware, orcombinations thereof.

Components, or modules, shown in block diagrams are illustrative ofexemplary embodiments of the invention and are meant to avoid obscuringthe invention. It shall also be understood that throughout thisdiscussion that components may be described as separate functionalunits, which may comprise sub-units, but those skilled in the art willrecognize that various components, or portions thereof, may be dividedinto separate components or may be integrated together, includingintegrated within a single system or component. It should be noted thatfunctions or operations discussed herein may be implemented ascomponents or modules.

Furthermore, connections between components within the figures are notintended to be limited to direct connections. Rather, data between thesecomponents may be modified, re-formatted, or otherwise changed byintermediary components (which may or may not be shown in the figure).Also, additional or fewer connections may be used. It shall also benoted that the terms “coupled” or “communicatively coupled” shall beunderstood to include direct connections, indirect connections throughone or more intermediary devices, and wireless connections.

In the detailed description provided herein, references are made to theaccompanying figures, which form a part of the description and in whichare shown, by way of illustration, specific embodiments of the presentinvention. Although these embodiments are described in sufficient detailto enable one skilled in the art to practice the invention, it shall beunderstood that these examples are not limiting, such that otherembodiments may be used, and changes may be made without departing fromthe spirit and scope of the invention.

Reference in the specification to “one embodiment,” “preferredembodiment,” “an embodiment,” or “embodiments” means that a particularfeature, structure, characteristic, or function described in connectionwith the embodiment is included in at least one embodiment of theinvention and may be in more than one embodiment. Also, such phrases invarious places in the specification are not necessarily all referring tothe same embodiment or embodiments. It shall be noted that the use ofthe terms “set” and “group” in this patent document may include anynumber of elements. Furthermore, it shall be noted that methods oralgorithms steps may not be limited to the specific order set forthherein; rather, one skilled in the art shall recognize, in someembodiments, that more or fewer steps may be performed, that certainsteps may optionally be performed, and that steps may be performed indifferent orders and may include some steps being done concurrently.

It shall also be noted that although embodiments described herein may bewithin the context of physicians or other caregivers, the inventionelements of the current patent document are not so limited. Accordingly,the invention elements may be applied or adapted for use in otherindustries and practices.

1. Overview of Physician Quality Scoring

FIG. 1 depicts various stages or factors 100 in a physician's trainingand career that may be used or considered when scoring a physician forquality or for a patient's specific needs according to embodiments ofthe present invention.

As shown in FIG. 1, one of the main stages in a physician's training ismedical school 105. Factors that may be considered relate to medicalschool include, but are not limited to, the school (e.g., HarvardMedical School) and its ranking, the years attended, physician'spersonal Medical College Admission Test (MCAT) score and/or Grade PointAverage (GPA), one or more grade point average (GPA) values and/or MCATscores of the physician's entering class, one or more grade pointaverage (GPA) values and/or MCAT scores of surrounding years' classes,and the like. For example, a medical student may have been accepted toHarvard Medical School and attended during the years 1986 through 1990.When gauging the quality of the training for that student, one or moremetrics (e.g., GPA, MCAT score, etc.) related to that student's enteringclass may be used. Furthermore, one or more metrics of surrounding classyears may also be used (e.g., that class year's GPA average, MCATaverage, current or past GPA average in medical school, etc.).

It shall be noted that one or more various measures of GPA, MCAT, orother scores may be used. For example, the average (mean, median, and/ormode), top X percentile, range, etc. may be used.

The next stage shown in FIG. 1 is internship 110. In the United States,a medical intern generally refers to someone who has or is working toobtain a medical degree but is not allowed to practice medicine withoutdirect supervision from someone who is fully licensed to practicemedicine. Not all physicians participate in an internship program duringthe course of their training. However, a physician participation in amedical internship (or their lack of participation) may be consideredwhen assessing the training quality of the physician.

Following medical school 105 or an internship 110, a medical schoolgraduate will enroll in a residency program 115. Medical school involvesmore academic endeavors whereas residency programs focus more on thepractical elements of the medical profession. Residency program may begeneral or directed to a specialty. For example, a person wanting tobecome a surgeon may become a surgical resident in a general surgicalpractice at a hospital.

Competition for residency programs can be fierce. Accordingly, theresidency program to which a medical school graduate is admitted may beused as an indicator in the quality of training. Also, an assessment ofthe peers accepted to that residency program can also reflect upon thequality of the physician.

FIG. 1 illustrates that a physician may have participated in more thanone residency 120. Each of these additional residency program orprograms 120 may also be considered when determining a physician'straining score.

The next stage shown in FIG. 1 is fellowship 125. A fellowship istypically an optional period of medical training or research focused ona certain specialty. A fellow may be a licensed physician who is capableof providing medical services to patients in the area in which they weretrained but have not yet qualified for that certain specialty. Aftercompleting a fellowship, a physician may provide medical services in thefellowship specialty without direct supervision of another physician.For example, as shown in FIG. 1, the physician may have done a residencyin general surgery at Brigham & Women's Hospital, but now desires tospecialize in thoracic surgery. To obtain the specialized training, thephysician may participate in a thoracic surgery fellowship. In someinstances, the fellowship may be heavily research based in which much ofthe fellow's time is spent in lab work or clinic trials.

Like residency, a physician may have participated in more than onefellowship program 130. Each of these additional fellowship program orprograms 130 may also be considered when determining a physician'straining score.

After formal medical training, a physician may have one or moreaffiliations. These affiliations represent practices at which thephysician may work or may have privileges. The quality of theseorganizations (e.g., Mercy Hospital) may be considered when scoring aphysician's training quality score. Also, the quality of the physiciansat the organizations may also be factored into the scoring. For example,the quality of doctors in the Surgical Associates group in Affiliation#2 shown in FIG. 1 may be considered when scoring the physician. If theSurgical Associates group comprises physicians with very goodcredentials (medical school, internships, residency programs, fellowshipprograms, other affiliates, etc.), this can help increase the score forthe physician of interest.

It shall be noted that one or more additional scoring factors may beconsidered when scoring a physician. These additional scoring factorsmay include, but are not limited to:

(1) Publication Track Record. A physician's publications may be usefulin scoring a physician. When considering publications, one or more ofseveral elements may be considered, including but not limited to:

-   -   (a) Subject matter covered in the publications;    -   (b) Number of article citations;    -   (c) Quality or scoring of co-authors;    -   (d) Frequency of co-authoring over time and author number (e.g.,        first or last author); and    -   (e) Trend(s) of publication volume and quality (e.g., impact        factor) over time.

(2) Physician Referrals. Physician referral may also be useful inscoring a physician. When considering physician referrals, one or moreof several elements may be considered, including by not limited to:

-   -   (a) Quality and/or specialty of physicians who refer patients to        the physician;    -   (b) Quality and/or specialty of physicians to whom the physician        refers patients to;    -   (c) Frequency of inbound and outbound referrals; and    -   (d) Concentration of inbound and outbound referrals.

(3) Volumes Data.

-   -   (a) Surgical procedures;    -   (b) Prescriptions;    -   (c) Tests;    -   (d) Diagnoses; and    -   (e) etc.

(4) Outcomes metrics.

-   -   (a) Survival rates;    -   (b) Complications rates;    -   (c) Readmissions rates; and    -   (d) etc.

(5) Honors & Awards. (e.g., Chief Resident, Alpha Omega Alpha,F.A.S.C.O., etc.)

(6) Professional Organization Memberships.

(7) Positions Held. (e.g., Dept. Chair, Board Examiner, Professor, etc.)

(8) Years of Experience.

(9) Wait Time to Soonest Appointment.

(10) Etc.

2. Mapping Residency & Fellowship Programs

FIG. 2 depicts relationships between medical schools, hospitals, andspecialty programs for which residency and fellowship programs may beoffered according to embodiments of the present invention. As shown inFIG. 2, most hospitals offer residency and fellowship programs fornumerous specialties (e.g., general surgery, pediatrics, dermatology,internal medicine, etc.). These programs may or may not be affiliatedwith a medical school. FIG. 2 depicts that the four hospitals (Hospital1 210—Hospital 4 225) are affiliated with XYZ Medical School 205. Inembodiments, the affiliations may be used in assessing quality of aprogram.

In embodiments, a residency or fellowship program quality may beassessed by the physicians attending a specific program and relatedprograms. For example, the quality of the General Surgery residency atHospital 230 is impacted by the residents who complete this program, aswell as those who complete other Hospital 230 residencies and other XYZMedical School-affiliated residencies. In embodiments, the weight ofthese relationships may vary by specialty. For example, General Surgeryand Orthopedic Surgery residents may have a disproportionate impact oneach other's program scores due to similarities in these programs. Inembodiments, other institutional factors, such as resources andrecognition, may also be considered.

3. Rating Medical School

In embodiments, quality of a physician's medical school (MS) may beassessed based on attributes of their peers, weighted by the proximityof those peers. An academic metric, such as average MCAT score and/orGPA score of an incoming class may be used. Attributes of previousand/or subsequent classes are also considered, but may be assigned alower weight. Institutional factors, such as NIH funding, may also beconsidered.

FIG. 3 graphically depicts the relationships between peers and medicalschool rating according to embodiments of the present invention. Inembodiments, a medical school 305 of a physician may be represented ashaving a set of one or more peers (e.g., box 340 may represent a singlepeer or a group of peers). In embodiments, each peer set may be weightedby proximity to the physician of interest. The proximity nexus may bebased upon one or more factors such as time or area of study. Given thatmost medical students at the same school have the same or vary similarcourse of study, a proximity factor may be based upon time (e.g., classyear).

FIG. 3 graphically depicts these temporal connections via the circles orrings. For example, the inner circle 310 represents peer groups thatwere the same entering class year as the physician of interest, and theouter circle 315 represents peer groups that were one year away (e.g.,one year prior, one year after, or both) from the entering class year asthe physician of interest. Only two groups 310 and 315 are depicted forsake of explanation, but it shall be noted that more or fewer groups maybe considered and that the temporal categories may represent variousranges of time.

As shown in FIG. 3, in embodiments, physicians who attended the samemedical school long before or long after the selected physician (e.g.,peer set 320) may have a lower proximity weighting as depicted by theweighting factor 330. Conversely, in embodiments, physicians (e.g.,physician set 325) who attended the same medical school within a shortertime period (e.g., the same year or within a few years) of the selectedphysician may be given more weight as graphically illustrated by theweighting factor 335.

As mentioned previously, in embodiments, the medical school score may bedetermined based, at least in part, upon one or more academic metricsand residency program scores of at least some of the physician's medicalschool peers. In embodiments, the medical school score may also be afunction of one or more institutional factors.

For example, in embodiments, the Medical School (MS) score may bedetermined as follows:

$\begin{matrix}{{MS} = {{{Physician}'}s\mspace{14mu} {Medical}\mspace{14mu} {School}{\mspace{11mu} \;}{Score}}} \\{= {f\left( {{MS}_{source},{MS}_{place},\; {{Institutional}{\mspace{11mu} \;}{{Factor}(s)}}} \right)}}\end{matrix}$e.g.,  = λ * MS_(source) + η * MS_(place) + θ * NIH  Funding

-   -   where:

MS_(source)=ƒ(MCAT,GPA)

e.g.,=α*MCAT_(median)+δ*GPA_(median)

MS_(place)=ƒ(Peers' Residency Program,Specialty,Time Attended)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{D_{i}\left( {{\alpha RP},{\mu \; S},{\delta \; T}} \right)}}}}$

-   -   where:        -   n=Number of physicians who attended the medical school        -   D_(i)( )=Individual physician's score        -   RP=A physician's residency        -   S=Specialty (e.g., General Surgery)        -   T=Time Attended (e.g., 1993)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{RP_{i}}}}}$

for all peers with the same specialty, S, who attended the programwithin X years of the physician

-   -   where:        -   n=Number of physicians who attended the medical school        -   RP_(i)=A physician's Residency Program score

4. Rating Residency Program

In embodiments, quality of a physician's residency program (RP) may beassessed based on attributes of their peers. For example, inembodiments, one or more of the following factors may be considered whendetermining the residency program quality: (1) the quality of medicalschools previously attended by these peers; (2) the quality offellowship programs subsequently attended; and (3) a peer's “proximity”to the physician of interest (e.g., closeness in time to when they werein the residency, whether they attended a different program at the sameinstitution or a similar program at an affiliated institution, etc.).

FIG. 4 graphically depicts the relationships between peers and residencyprogram rating according to embodiments of the present invention. Inembodiments, a residency program 405 of a physician may be representedas having a set of one or more peers (e.g., box 440 may represent asingle peer or a group of peers). In embodiments, each peer set may beweighted by proximity to the physician of interest. The proximity nexusmay be based upon one or more factors such as time, institution,residency specialty, etc.

FIG. 4 graphically depicts these connections via the circles or rings.For example, the inner circle 410 represents physicians who completedthe same residency within a few years of the selected physician. And, inembodiments, the outer circle 415 represents peer groups that were inthe same residency program but at a different time period, that were ina different program at the same institution, or that were in a similarprogram at an affiliated institution. Only two groups 410 and 415 aredepicted for sake of explanation, but it shall be noted that more orfewer groups may be considered and that the categories may representvarious factors or various combinations of factors as suggested above.

As shown in FIG. 4, in embodiments, physicians who have a close nexus tothe selected physician (e.g., peer set 425) may have a higher proximityweighting as depicted by the weighting factor 435. Conversely, inembodiments, physicians (e.g., physician set 420) who do not have asclose a nexus to the selected physician may be given less weight asgraphically illustrated by the weighting factor 430.

In embodiments, a residency program score for the physician may bedetermined based, at least in part, upon one or more incoming peerattributes (e.g., medical school scores) and one or more outgoing peerattributes (e.g., fellowship program scores) of at least some of thephysician's peers in one or more residency programs. In embodiments, theresidency program score may also be a function of one or moreinstitutional factors.

For example, in embodiments, the Residency Program (RP) score may bedetermined as follows:

$\begin{matrix}{{RP} = {{{Physician}'}s\mspace{14mu} {Residency}\mspace{14mu} {Program}\mspace{14mu} {Score}}} \\{= {f\left( {{RP}_{source},{RP}_{place},{{Institutional}\mspace{14mu} {{Factor}(s)}}} \right)}}\end{matrix}$e.g.,  = λ * RP_(source) + η * RP_(place) + θ * NIH  Funding

-   -   where:

RP_(source)=ƒ(Peer's Medical School,Time Attended)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{C_{i}\left( {{\alpha \; {MS}},{\mu \; S},{\delta \; T}} \right)}}}}$

-   -   where:        -   n=Number of physicians who attended the residency program        -   C_(i)( )=Individual physician's score        -   MS=A physician's medical school        -   S=Specialty (e.g., General Surgery)        -   T=Time Attended (e.g., 1993)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{MS}_{i}}}}$

for all peers with the same specialty, S, who attended the programwithin X years of the physician

-   -   where:        -   n=Number of physicians who attended the residency program        -   MS_(i)=A physician's Medical School score

RP_(place)=ƒ(Peers' Fellowship Program,Time Attended)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{D_{i}\left( {{\alpha \; {FP}},{\mu \; S},{\delta \; T}} \right)}}}}$

-   -   where:        -   n=Number of physicians who attended the residency program        -   D_(i)( )=Individual physician's score        -   FP=Fellowship Program (e.g., Steadman Hawkins)        -   S=Specialty (e.g., General Surgery)        -   T=Time Attended (e.g., 1993)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{FP}_{i}}}}$

for all peers with the same specialty, S, who attended the programwithin X years of the physician

-   -   where:        -   n=Number of physicians who attended the residency program        -   FP_(i)=A physician's Fellowship Placement score

5. Rating Fellowship Program

In embodiments, quality of a physician's fellowship program (FP) may beassessed based on attributes of their peers. For example, inembodiments, one or more of the following factors may be considered whendetermining the fellowship program quality: (1) the quality of residencyprograms previously attended by these peers; (2) the quality of theinstitutions where they subsequently practice; and (3) a peer's“proximity” to the physician of interest (e.g., closeness in time towhen they were in the fellowship, whether they attended a differentprogram at the same institution or a similar program at an affiliatedinstitution, etc.). In embodiments, institutional factors, such aspublication track record, may also be considered.

FIG. 5 graphically depicts the relationships between peers andfellowship program rating according to embodiments of the presentinvention. In embodiments, a fellowship program 505 of a physician maybe represented as having a set of one or more peers (e.g., box 540 mayrepresent a single peer or a group of peers). In embodiments, each peerset may be weighted by proximity to the physician of interest. Theproximity nexus may be based upon one or more factors such as time,institution, fellowship specialty, etc.

FIG. 5 graphically depicts these connections via the circles or rings.For example, the inner circle 510 represents physicians who completedthe same fellowship within a few years of the selected physician. And,in embodiments, the outer circle 515 represents peer groups that were inthe same fellowship program but at a different time period, that were ina different program at the same institution, or that were in a similarprogram at an affiliated institution. Only two groups 510 and 515 aredepicted for sake of explanation, but it shall be noted that more orfewer groups may be considered and that the categories may representvarious factors or various combinations of factors as suggested above.

As shown in FIG. 5, in embodiments, physicians who have a close nexus tothe selected physician (e.g., peer set 525) may have a higher proximityweighting as depicted by the weighting factor 535. Conversely, inembodiments, physicians (e.g., physician set 520) who do not have asclose a nexus to the selected physician may be given less weight asgraphically illustrated by the weighting factor 530.

In embodiments, a fellowship program score for the physician may bedetermined based, at least in part, upon one or more incoming peerattributes (e.g., residency program scores) and one or more outgoingpeer attributes (e.g., practice groups/locations scores) of at leastsome of the physician's peers in one or more fellowship programs. Inembodiments, the fellowship program score may also be a function of oneor more institutional factors.

For example, in embodiments, the Fellowship Program (FP) score may bedetermined as follows:

$\begin{matrix}{{RP} = {{{Physician}'}s\mspace{14mu} {Fellowship}\mspace{14mu} {Program}\mspace{14mu} {Score}}} \\{= {f\left( {{FP}_{source},{FP}_{place},{{Institutional}\mspace{14mu} {{Factor}(s)}}} \right)}}\end{matrix}$e.g.,  = λ * FP_(source) + η * FP_(place) + θ * NIH  Funding

-   -   where:

FP_(source)=ƒ(Peers' Residency Program,Time Attended)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{C_{i}\left( {{\alpha \; {RP}},{\mu \; S},{\delta \; T}} \right)}}}}$

-   -   where:        -   n=Number of physicians who attended the fellowship program        -   C_(i)( )=Individual physician's score        -   RP=Residency Program (e.g., Hospital for Special Surgery)        -   S=Specialty (e.g., General Surgery)        -   T=Time Attended (e.g., 1993)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{RP}_{i}}}}$

for all peers with the same specialty who attended the program within Xyears of the physician

-   -   where:        -   n=Number of physicians who attended the fellowship program        -   RP, =A physician's Residency Program score

FP_(place)=ƒ(Peers' Practice Location,Time Attended)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{D_{i}\left( {{\alpha \; {PL}},{\mu \; S},{\delta \; T}} \right)}}}}$

-   -   where:        -   n=Number of physicians who attended the fellowship program        -   D_(i)( )=Individual physician's score        -   PL=Practice Location/Group        -   S=Specialty (e.g., General Surgery)        -   T=Time Attended (e.g., 1993)

${e.g.},{= {\frac{1}{n}{\sum\limits_{i = 0}^{n}{PL}_{i}}}}$

for all peers with the same specialty who attended the program within Xyears of the physician

-   -   where:        -   n=Number of physicians who attended the fellowship program        -   PL=A physician's Practice Location/Group score

In embodiments, the parameter weightings for any of the above-listedcalculations (e.g., α, μ, δ) may be determined programmatically. Inembodiments, initial values may be assigned to all parameters. Theweights may then be sequentially adjusted through iterations in order tominimize the mean difference between the quality rating for each step ofa physician's training (med school, residency, fellowship training). Inembodiments, as a default, the weights may be set to assign zero weightto all physicians who did not attend the same school and/or specialty asthe physician and assign equal non-zero weight to all physicians whoattended the same program, regardless of time attended. Determining aPhysician's Overall Training Score

Turning now to FIG. 6, depicted is a methodology for assigning anoverall training score to a physician according to embodiments of thepresent invention. As shown in FIG. 6, an initial step is to initialize(605) values. For example, for the first iteration since Medical School(MS) scores are based upon Residency Program (RP) scores, which have notyet been calculated, the RP score may be set to an initial value orvalues. In addition, in embodiments, the initialization step may beconsidered to include compiling the initial raw data values, such asMCAT and GPA values, years and location of residency program(s), yearsand location of fellowship program(s), etc. This information may bestored in one or more storage devices and accessed by one or moreprocessors.

Having gathered the raw data and initialized values, Medical School (MS)scores for a set of one or more physicians may be calculated (610) basedupon attributes of medical students and Residency Program scores. Inembodiments, the MS score may be calculated as discussed above in whichMS=ƒ(MS_(source), MS_(place), Institutional Factor(s)).

Having calculated the Medical School scores, Residency Program (RP)scores may be calculated (615) based upon the Medical School scores thatwere just calculated and Fellowship Programs scores. In embodiments, theRP score may be calculated as discussed above in which RP=ƒ(RP_(source),RP_(place), Institutional Factor(s)).

Having calculated the Residency Program scores, Fellowship Program (FP)scores may be calculated (620) based upon the Residency Program scoresthat were just calculated and Practice Location/Group scores. Inembodiments, the FP score may be calculated as discussed above in whichFP=ƒ(FP_(source), FP_(place), Institutional Factor(s)).

In embodiments, the process of assigning a physician's training scoremay be obtained by iterating the above steps until a stop condition hasbeen reached. In embodiments, a stop condition may be considered to havebeen reached when a correlation (or correlations) between physicians'medical school, residency program, and fellowship quality scores ismaximized Thus, in embodiments, one or more correlation factors may becalculated (625) using the MS, RP, and FP scores in order to determineif the process should stop or be iterated (630).

FIG. 7 depicts a methodology for determining a correlation factor aspart of the iteration process according to embodiments of the presentinvention. In embodiments, for each iteration, two coefficients ofdetermination are computed. A first coefficient of determination iscalculated (705) where physicians' Residency Program scores are assumedto be a linear function of their Medical School scores. For example, inembodiments, the first coefficient of determination may be computed asfollows:

${{R^{2}\left( {{RP},{MS}} \right)} = {1 - \frac{\sum\limits_{i = 0}^{n}\left( {{RP} - {MS}} \right)^{2}}{\sum\limits_{i = 0}^{n}\left( {{RP} - \overset{\_}{RP}} \right)^{2}}}},$

for all n physicians.

A second coefficient of determination is calculated (710) wherephysicians' Fellowship Program scores are assumed to be a linearfunction of their Residency Program scores. For example, in embodiments,the second coefficient of determination may be computed as follows:

${{R^{2}\left( {{FP},{RP}} \right)} = {1 - \frac{\sum\limits_{i = 0}^{n}\left( {{FP} - {RP}} \right)^{2}}{\sum\limits_{i = 0}^{n}\left( {{FP} - \overset{\_}{FP}} \right)^{2}}}},$

for all n physicians.

The coefficients may then be added together. In embodiments, greaterweight may be placed on the correlation between residency and fellowshipquality because residency performance is typically considered a betterindicator of true quality than medical school performance. Inembodiments, the coefficients may be combined together to form acorrelation factor, σ, as follows:

σ=R ²(RP,MS)+γ×R ²(FP,RP)

Thus, in embodiments, an objective of the iterative scoring is tomaximize σ across all physicians. When σ is maximized, a stop conditionis considered to be reached.

In embodiments, a number of stop conditions may be set. For example, astop condition may be when a difference between the correlation factorfor a current iteration and the correlation factor of a prior iterationis below a threshold. Another stop condition may be if the correlationfactor starts to diverge (e.g., if the correlation factor for a currentiteration is less than the correlation factor of a prior iteration).Also, a stop condition may be if a set number of iterations has beenreached. One skilled in the art shall recognize that there are numberways of performing iterative calculations (including setting stopconditions), which may be employed herein.

It shall be noted that, in embodiments, in addition to iterating thetraining scoring process, the coefficients for each parameter may bemodified. In embodiments, an objective is to set the optimal weightingsso that the scoring iterations achieve the absolute minimum solution,rather than a local minimum.

In embodiments, to achieve optimal weightings, the process is startedwith a simple set of weights, which are then systematically experimentedwith by altering these values. Consider, by way of illustration, thefollowing example methodology:

Step #1—set the initial coefficients:

-   -   α, the coefficient for program quality (MS, RP, FP) may be set        to 1 for all programs;    -   μ and δ, the coefficients for specialty, S, and time attended,        T, may be set to 0 for all physicians;    -   λ and η, the coefficients for programs' sourcing and placement        quality, may be set to 0.5 for all programs; and    -   θ, the coefficient for NIH funding (or some other institutional        factor or factors), may be set to 0.

Step #2—adjust the specialty coefficients. In embodiments, μ may beincrementally increased until σ no longer decreases with each increase;this may be done for one or more specialty at a time to account for thefact that the optimal coefficient may vary by specialty.

Step #3—adjust the time attended coefficients. In embodiments, δ may beincrementally increased until δ no longer decreases with each increase;this may be done for one or more specialty at a time to account for thefact that the optimal coefficient may vary by specialty.

Step #4—adjust other institutional factor coefficients. In embodiments,θ may be increased incrementally until δ no longer decreases with eachincrease; this too may be done for one or more specialty at a time toaccount for the fact that the optimal coefficient may vary by specialty.

In embodiments, the physician's training programs may be scorediteratively with different combinations of parameter coefficients untilan absolute minimum for 6 is achieved.

Returning to FIG. 6, once a stop condition has been reached (630), inembodiments, a physician's overall training score may be computed usingthe physician's final Medical School, Residency Program, and FellowshipProgram scores. In embodiments, a composite quality score of aphysician's training may be determined as follows:

$\begin{matrix}{{MD}_{train} = {{Composite}\mspace{14mu} {training}\mspace{14mu} {quality}\mspace{14mu} {score}}} \\{= {f\left( {{MS},{RP},{FP}} \right)}} \\{= {{\alpha \; {MS}} + {\mu \; {RP}} + {\delta \; {FP}}}}\end{matrix}\quad$

-   -   where:        -   MS=Medical School score        -   RP=Residency Program score        -   FP=Fellowship Program score

In embodiments, the coefficients for medical school, residency, andfellowship scores may be calibrated against other external indicators ofphysician quality. FIG. 8 depicts a method for assigning a physician'soverall training using correlations according to embodiments of thepresent invention.

In embodiments, the initial coefficients may be based (805) on physiciansurvey data. Typically, physicians place the greatest weight on theirpeer's fellowship training, the second greatest weight on residencytraining, and the least weight on medical school attended. Thus, inembodiments, to approximate these preferences, the coefficients may beset as follows: α=0.2; μ=0.35; and δ=0.45—although it shall be notedthat other values may be set.

Then, in embodiments, correlations may be calculated (810) withindicators of academic quality. At academic centers, two indicators ofphysician quality are: (1) positions held, and (2) publication trackrecord. Academic physicians may be first rated based on the number ofpositions held with certain titles (e.g., chief, head, or director).They may also be rated based on volume and quality of publications, asmeasured by the impact factor of the publishing journal

$\left( {{e.g.},{\sum\limits_{i = 0}^{n}J_{i}},} \right.$

where n=number of publications, J=journal's impact factor).

In embodiments, the coefficients of determination, R², for each trainingvariable and measure of academic quality may then be calculated:

TABLE A Medical Residency Fellowship School Score Program Score ProgramScore Positions Held Score R² _(MS, PH) R² _(RP, PH) R² _(FP, PH)Publication Score R² _(MS, P) R² _(RP, P) R² _(FP, P)

In embodiments, the correlations with indicators of clinical quality arealso calculated (815). Clinical quality may be ascertained fromphysician's outcomes data (e.g., mortality rates, readmission rates,complication rates, etc.), peer opinion, and preferred clinicalpractices, among other factors. In embodiments, to calibrate thetraining quality measures, specialties in which outcomes data and peeropinion are likely to be accurate indicators of true clinical qualitymay be focused upon. Such specialties may include cardiothoracicsurgery, cardiology, oncology, neurosurgery, and orthopedic surgery.Outcomes data may be based on published indicators of physicianperformance. For example, what percent of patients are readmitted to thehospital within 30 days of receiving a knee replacement from a givenorthopedic surgeon?

In embodiments, peer opinion may be obtained by surveying physiciansabout their peers in the same specialty and geographic region (e.g.,other thoracic surgeons in the same state). Physicians may be asked toidentify which of their peers they would recommend to patients if they,themselves, were unable to see the patient or who they would select astheir doctor.

In embodiments, preferred clinical practices may be inferred fromprovider-level claims data. This analysis may focus on specificprocedures or treatments where many physicians are not treating patientsaccording to the latest recommended guidelines. For example, the besturologists treating renal cell carcinoma will conduct three partialnephrectomies for every full nephrectomy; however, many urologists stilldefault to the old standard of conducting full nephrectomies in amajority of patients.

In embodiments, coefficients of determination, R², may then calculatedfor each training variable and each clinical performance measure:

TABLE B Medical Residency Fellowship School Score Program Score ProgramScore Outcomes Data Score R² _(MS, O) R² _(RP, O) R² _(FP, O) PeerOpinion Score R² _(MS, PO) R² _(RP, PO) R² _(FP, PO) Clinic PracticeScore R² _(MS, CP) R² _(RP, CP) R² _(FP, CP)

Given the various coefficients, final coefficients may be determined(820). In embodiments, an average of the coefficients of determinationmay be used to calculate the final coefficients for medical school,residency, and fellowship. Note that the academic quality indicators areonly included for physicians who practice at academic institutions.

TABLE C Medical Residency Fellowship School Score Program Score ProgramScore Positions Held Score R² _(MS, PH) R² _(RP, PH) R² _(FP, PH)Publication Score R² _(MS, P) R² _(RP, P) R² _(FP, P) Outcomes DataScore R² _(MS, O) R² _(RP, O) R² _(FP, O) Peer Opinion Score R²_(MS, PO) R² _(RP, PO) R² _(FP, PO) Clinic Practice Score R² _(MS, CP)R² _(RP, CP) R² _(FP, CP)

For example, in embodiments, a, the coefficient for medical school maybe set to equal:

For academics:α=(R ² _(MS,P) +R ² _(MS,P) +R ² _(MS,P) +R ² _(MS,P) +R ²_(MS,P))/[Sum of all R ²]

For non-academics:α=(R ² _(MS,P) +R ² _(MS,P) +R ² _(MS,P))/[Sum of allclinical R ²]

It shall be noted that, in embodiments, the denominator equals the sumof all R² for medical school, residency, and fellowship scores. It shallalso be noted that, in embodiments, additional coefficients may be addedto this equation to place greater weight on certain clinical or qualityindicators. The example above reflects a straight average that assignsequal weight to each indicator.

6. Rating the Physician's Past and Current Practice Groups/Locations

In embodiments, quality of a physician's post-training practicegroups/locations (P) may be determined by the quality of their peers ateach practice. It shall be noted that practice location may meanpractice group (including doctors who work in a small group, in the samedepartment, in the same team, etc.), physicians working for the sameorganization (e.g., physicians in the same department, in the samehospital, in the same organization, etc.), even if the physicians arenot at the same physical location.

FIG. 9 graphically depicts the relationships between peers and practicelocations/groups according to embodiments of the present invention. Inembodiments, a practice location 905 of a physician may be representedas having a set of one or more peers (e.g., box 940 may represent asingle peer or a group of peers).

In embodiments, peer quality may be determined by the quality ofphysicians' overall training. In alternative embodiments, peer qualitymay also be a function of one or more additional factors, such as (byway of example and not limitation), publications, outcomes data, honors& awards, positions held, and the patient referrals they receive fromother physicians.

In embodiments, each peer set may be weighted by “proximity” to thephysician of interest. FIG. 9 graphically depicts these “proximity”connections via the circles or rings. For example, the inner circle 910represents physicians who work in the same department and at the sametime as the selected physician. And, in embodiments, the outer circle915 represents peer groups that were in the same department but at adifferent time period or that were in different programs at the sameinstitution. Only two groups 910 and 915 are depicted for sake ofexplanation, but it shall be noted that more or fewer groups may beconsidered and that the categories may represent various factors orvarious combinations of factors.

In embodiments, a practice group or location score quality may beweighted by peer's “proximity,” as determined by one or more nexusfactors, such as (by way of example and not limitation), when theyworked at the practice location, whether they worked for the samedepartment or a related department, and how much time they spent at thatpractice location. In embodiments, disproportionate weight may beassigned to the top physicians at each practice location.

As shown in FIG. 9, in embodiments, physicians who have a close nexus tothe selected physician (e.g., peer set 925) may have a higher proximityweighting as depicted by the weighting factor 935. Conversely, inembodiments, physicians (e.g., physician set 920) who do not have asclose a nexus to the selected physician may be given less weight asgraphically illustrated by the weighting factor 930.

In embodiments, the Practice Group/Location (P) score may be determinedas follows:

$\begin{matrix}{P = {{Practice}\mspace{14mu} {{Location}/{Group}}\mspace{14mu} {Score}}} \\{\left( {{e.g.},{{SF}\mspace{14mu} {General}\mspace{14mu} {Hospital}},{Cardiology},2014} \right)} \\{= {f\left( {{{{Peers}'}\mspace{14mu} {Quality}},{Proximity}} \right)}}\end{matrix}$${e.g.} = \frac{\sum\limits_{i = 1}^{n}{{MD}_{{train}_{i}}*\left( \frac{1}{1 + {\mu \; i}} \right)*{D_{i}\left( {{Dept},{Years}} \right)}}}{\sum\limits_{i = 1}^{n}{\left( \frac{1}{1 + {\mu \; i}} \right)*{D_{i}\left( {{Dept},{Years}} \right)}}}$

-   -   where:    -   n=Number of physicians who have worked at the practice location        (e.g., SF General Hospital)    -   MD_(train)=Quality of physician's training program    -   D_(i)( )=Proximity of the physician to the practice group        -   e.g., D_(i)( )=1 if same department and practiced there at            the same time        -   D_(i)( )=0 if different department or practiced there at            different time

In embodiments, physicians at each practice group are ranked indescending order by training quality score so that the greatest weightis placed on the top physicians at the practice.

In embodiments, μ may be calibrated based on peer ratings of topacademic institutions around the country. An academic experts panel maybe asked to identify the top 5 institutions for their medical specialty.μ may then be adjusted to maximize the R² between the algorithm'sratings of the top 10 academic institutions in each specialty and thenumber of votes received from the panelists.

7. Determining A Physician's Overall Quality Score

FIG. 10 depicts a method for determining a physician's overall qualityscore according to embodiments of the present invention. As depicted, aphysician's training score, which may be based, at least in part, uponquality of the physician's peers, is determined (1005). This trainingscore may be determined as described above.

Also, in embodiments, a rating for the physician's practicegroup/location, which may be based, at least in part, upon quality ofthe physician's peers at the practice group/location, is determined(1010). This score may be determined as described above.

Given a physician's training score and a physician's practice score, thephysician's overall quality score may be assigned (1015) to thephysician based, at least in part, upon those values. In embodiments,the overall quality score of a physician may be calculated as a weightedaverage of the physician's training quality score and the averagequality score of their practice groups.

MD_(quality)=αMD_(train) +μP

-   -   where μP=mean quality score of a physician's practice        groups/locations

In embodiments, as a default, equal weight may be assigned to bothcoefficients, a and μ. In alternative embodiments, disproportionalweight may be placed on the highest scoring practice groups a physicianis affiliated with.

8. Using a Physicians' Scoring

Having assigned a physician's overall quality score, this informationmay be used in various ways. For example, in embodiments, a patient mayuse this information to help identify a physician.

In embodiments, a patient may use this information to help identifywhich physician is the best “fit” for him or her to provide care. Inembodiments, “fit” may be determined not only by the physician's overallquality score but may also be based on, or weighted against, variousfactors including, but not limited to, the physician's specific area ofsub-specialty training, stated clinical interests, volume of clinicalexperience, distance from the patient, appointment availability, andpast patient satisfaction scores. One skilled in the art shall recognizethat other factors, weights, and matching methods may be employed toalign a patient with the best qualified doctor.

9. Computing System Embodiments

Having described the details of the invention, an exemplary system 1100,which may be used to implement one or more of the methodologies of thepresent invention, will now be described with reference to FIG. 11. Asillustrated in FIG. 11, the system includes a central processing unit(CPU) 1101 that provides computing resources and controls the computer.The CPU 1101 may be implemented with a microprocessor or the like, andmay also include a graphics processor and/or a floating pointcoprocessor for mathematical computations. The system 1100 may alsoinclude system memory 1102, which may be in the form of random-accessmemory (RAM) and read-only memory (ROM).

A number of controllers and peripheral devices may also be provided, asshown in FIG. 11. An input controller 1103 represents an interface tovarious input device(s) 1104, such as a keyboard, mouse, or stylus.There may also be a scanner controller 1105, which communicates with ascanner 1106. The system 1100 may also include a storage controller 1107for interfacing with one or more storage devices 1108 each of whichincludes a storage medium such as solid state drives, magnetic tape ordisk, or an optical medium that might be used to record programs ofinstructions for operating systems, utilities and applications which mayinclude embodiments of programs that implement various aspects of thepresent invention. Storage device(s) 1108 may also be used to storeprocessed data or data to be processed in accordance with the invention,including data for determining a physician's score(s). FIG. 12 depictsat least some datastores that may be used in assessing a physicians'score(s) or ranking(s) according to embodiments of the presentinvention. The system 1100 may also include a display controller 1109for providing an interface to a display device 1111, which may be acathode ray tube (CRT), a thin film transistor (TFT) display, or othertype of display. The system 1100 may also include a printer controller1112 for communicating with a printer 1113. A communications controller1114 may interface with one or more communication devices 1115, whichenables the system 1100 to connect to remote devices through any of avariety of networks including the Internet, a local area network (LAN),a wide area network (WAN), or through any suitable electromagneticcarrier signals including infrared signals.

In the illustrated system, all major system components may connect to abus 1116, which may represent more than one physical bus. However,various system components may or may not be in physical proximity to oneanother. For example, input data and/or output data may be remotelytransmitted from one physical location to another. In addition, programsthat implement various aspects of this invention may be accessed from aremote location (e.g., a server) over a network. Such data and/orprograms may be conveyed through any of a variety of machine-readablemedium including magnetic tape or disk or optical disc, or atransmitter, receiver pair.

Embodiments of the present invention may be encoded upon one or morenon-transitory computer-readable media with instructions for one or moreprocessors or processing units to cause steps to be performed. It shallbe noted that the one or more non-transitory computer-readable mediashall include volatile and non-volatile memory. It shall be noted thatalternative implementations are possible, including a hardwareimplementation or a software/hardware implementation.Hardware-implemented functions may be realized using ASIC(s),programmable arrays, digital signal processing circuitry, or the like.Accordingly, the “means” terms in any claims are intended to cover bothsoftware and hardware implementations. Similarly, the term“computer-readable medium or media” as used herein includes softwareand/or hardware having a program of instructions embodied thereon, or acombination thereof. With these implementation alternatives in mind, itis to be understood that the figures and accompanying descriptionprovide the functional information one skilled in the art would requireto write program code (i.e., software) and/or to fabricate circuits(i.e., hardware) to perform the processing required.

While the inventions have been described in conjunction with severalspecific embodiments, it is evident to those skilled in the art thatmany further alternatives, modifications, application, and variationswill be apparent in light of the foregoing description. Thus, theinventions described herein are intended to embrace all suchalternatives, modifications, applications and variations as may fallwithin the spirit and scope of the appended claims.

1-20. (canceled)
 21. The quality scoring system comprising: a memory;one or more processors configured to cause the quality scoring systemto: obtain, from one or more data sources, one or more data setsassociated with a plurality of individuals; determine a first set oftraining values of a plurality of training values associated with anindividual of the plurality of individuals, wherein the first set oftraining values are based on a seed value related to the individual'straining and are iteratively determined using a series of correlationfactors based on a plurality of training values that are prioriterations of the first set of training values and a condition evaluatedon the series of correlation factors, wherein determination of first setof training values of the individual comprises: determining eachcorrelation factor of the series of correlation factors using first andsecond coefficients that are each associated with a plurality ofdifferent types of training values; and applying the series ofcorrelation factors to training values of the plurality of individuals;determine a practice location value associated with the individual basedon practice-location attributes of at least some of the plurality ofindividuals; and determine a quality score of the individual using thepractice location value and the first set of training values.
 22. Thequality scoring system of claim 21, wherein the first coefficient isdetermined using a first type of training value and a second type oftraining value.
 23. The quality scoring system of claim 22, wherein thesecond coefficient is determined using the second type of training valueand a third type of training value.
 24. The quality scoring system ofclaim 21, wherein the first set of training values are determined basedon a modification of the first and second coefficients.
 25. The qualityscoring system of claim 21, wherein the first and second coefficientsare averaged to determine final coefficients that are used to determinethe first set of training values.
 26. The quality scoring system ofclaim 21, wherein the condition evaluated on the series of correlationfactors is difference between a present correlation factor of the seriesof correlation factors and a prior correlation factor of the series ofcorrelation factors is below a threshold.
 27. The quality scoring systemof claim 21, wherein the condition evaluated on the series ofcorrelation factors is whether a present correlation factor of theseries of correlation factors is less than a prior correlation factor ofthe series of correlation factors.
 28. The quality scoring system ofclaim 21, wherein the condition evaluated on the series of correlationfactors is completion of a predetermined number of iterations.
 29. Thequality scoring system of claim 21, wherein the training values includea first type of training value, a second type of training value, andthird type of training value; wherein the first type of training valueis determined using attributes of the at least some of the plurality ofindividuals and a second type of training value, the second type oftraining value is determined using attributes of the at least some ofthe plurality of individuals and the first type of training value and athird type of training value, and the third type of training value isdetermined using attributes of the at least some of the plurality ofindividuals and the second type of training value and the practicelocation value.
 30. The quality scoring system of claim 29, wherein thefirst type of training value is a medical school score, the second typeof training value is a residency program score, and the third type oftraining value is a fellowship program score.
 31. The quality scoringsystem of claim 21, wherein the set of instructions that is executableby the computing device to cause the computing device to perform:displaying, to a user, information based on the determined quality scoreof the individual.
 32. The quality scoring system of claim 31, whereinthe displayed information is based on a request for information aboutthe individual.
 33. A method performed by a quality scoring system forcalculating quality score of an individual, the method comprising:obtaining, from one or more data sources, one or more data setsassociated with a plurality of individuals; determining a first set oftraining values of a plurality of training values associated with anindividual of the plurality of individuals, wherein the first set oftraining values are based on a seed value related to the individual'straining and are iteratively determined using a series of correlationfactors based on a plurality of training values that are prioriterations of the first set of training values and a condition evaluatedon the series of correlation factors, wherein determination of first setof training values of the individual comprises: determining eachcorrelation factor of the series of correlation factors using first andsecond coefficients that are each associated with a plurality ofdifferent types of training values; and applying the series ofcorrelation factors to training values of the plurality of individuals;determining a practice location value associated with the individualbased on practice-location attributes of at least some of the pluralityof individuals; and determining a quality score of the individual usingthe practice location value and the first set of training values. 34.The method of claim 33, wherein the first coefficient is determinedusing a first type of training value and a second type of trainingvalue.
 35. The method of claim 34, wherein the second coefficient isdetermined using the second type of training value and a third type oftraining value.
 36. The method of claim 33, wherein the first set oftraining values are determined based on a modification of the first andsecond coefficients.
 37. The method of claim 33, wherein the conditionevaluated on the series of correlation factors is difference between apresent correlation factor of the series of correlation factors and aprior correlation factor of the series of correlation factors is below athreshold.
 38. The method of claim 33, wherein the condition evaluatedon the series of correlation factors is whether a present correlationfactor of the series of correlation factors is less than a priorcorrelation factor of the series of correlation factors.
 39. The methodof claim 33, wherein the training values include a first type oftraining value, a second type of training value, and third type oftraining value; wherein the first type of training value is determinedusing attributes of the at least some of the plurality of individualsand a second type of training value, the second type of training valueis determined using attributes of the at least some of the plurality ofindividuals and the first type of training value and a third type oftraining value, and the third type of training value is determined usingattributes of the at least some of the plurality of individuals and thesecond type of training value and the practice location value.
 40. Themethod of claim 33, wherein the set of instructions that is executableby the computing device to cause the computing device to perform:displaying, to a user, information based on the determined quality scoreof the individual.